Job Title: AI Engineer / Generative AI EngineerLocation: Los Angeles, CA (Hybrid)
Duration: 3+ Months (Extension Possible)
Experience: 15+ YearsJob SummaryWe are seeking a hands-on AI Engineer with expertise in Python, Agentic AI, AWS Bedrock AgentCore, and LLM evaluation to build, deploy, monitor, and continuously improve production-grade AI agents.Key ResponsibilitiesDevelop AI agents using Python and Claude Agent SDK.Deploy agents using AWS Bedrock AgentCore Runtime.Design golden datasets and implement LLM-as-a-Judge evaluations.Integrate automated evaluations into CI/CD pipelines as quality gates.Implement Langfuse/OpenTelemetry for AI observability and tracing.Evaluate agent accuracy, task completion, tool execution, and performance.Support RAG pipelines and AWS data processing workflows.Required SkillsPython and Agentic AI development.AWS Bedrock AgentCore Evaluations / Bedrock evaluation.Golden-dataset design and curation.LLM-as-a-Judge design and calibration against human labels.Automated evaluation and CI gate integration.Langfuse and/or OpenTelemetry.Good to HaveRAGAS or equivalent evaluation frameworks.Trajectory evaluation and agentic red-teaming.Bedrock Guardrails, Strands Agents SDK, and GitHub Actions.Qualifications6–8+ years of software engineering, cloud engineering, or AI/ML experience.Hands-on experience with production AI agents and continuous evaluation.Strong problem-solving and troubleshooting skills.
